Within the field of computing, many scenarios involve the application of evaluation techniques to a data set in pursuit of a goal. For example, the evaluation techniques may comprise machine learning models, such as artificial neural networks, statistical classifiers, genetically generated processes, and adaptive finite state machines that, when applied to a data set, perform various types of evaluation on the data set, such as classifying data units into similar classes; ranking the respective data units of the data set; identifying patterns arising within the data set; and performing a search over the data set.
In many such scenarios, the evaluation of the data set is performed by a designer who chooses a suitable evaluator and configures it in a manner that is suitable for the data set and the goal. The designer may apply a training routine to the evaluator with a training data set that reflects the goal (e.g., a set of inputs with known-correct outputs), and may iteratively train the evaluator until the evaluator is capable of achieving the goal over the provided training data within a desired degree of confidence. After verifying the suitability of the evaluator, the designer may apply the evaluator to the data set (e.g., in a production environment where the correct output for various input is occasionally unknown), and may take the evaluation results of the evaluator as output that expresses the goal over the data set. In this manner, an evaluator may be designed to produce desired results over a data set, even for data for which the correct output values are unknown and/or difficult for humans to evaluate and choose output values in a consistent manner.